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FOREWORD 


This  report  describes  the  development  and  evaluation  of  an  officer  potential  composite  that 
could  expand  the  scope  of  the  United  States  Naval  Academy  (USNA)  selection  system  to  include 
the  prediction  of  officer  performance.  Officer  fitness  reports  were  used  to  develop  a  criterion 
measure  that  permitted  a  performance  score  to  be  assigned  to  each  officer.  In  turn,  newly 
developed  measures  were  combined  with  existing  valid  measures  based  on  information  available 
at  the  time  of  application  to  the  Naval  Academy  to  construct  and  evaluate  a  composite  to  predict 
this  measure  of  officer  performance. 

This  effort  was  conducted  under  the  sponsorship  of  the  Office  of  Naval  Technology  within 
exploratory  development  (Program  Element  0602233N  Project  Task  RM33M20.05  Integrating 
Officer  Selection  Systems)  in  response  to  a  request  by  the  USNA  to  develop  a  measure  of  officer 
potential  in  the  selection  of  future  midshipmen. 


B.  E.  BACON 
Captain,  U.S.  Navy 
Commanding  Officer 


J.S.MCMICHAEL 
Technical  Director 
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SUMMARY 


Problem 

Current  United  States  Naval  Academy  (USNA)  selection  procedures  are  quite  successful  in 
identifying  applicants  with  the  greatest  potential  for  successful  performance  during  training.  These 
procedures,  which  include  a  variety  of  selection  factors  related  to  academy  success,  have  been 
combined  into  a  single  composite  measure,  the  Candidate  Multiple  (CM),  to  serve  as  the  primary 
tool  for  evaluating  candidates.  Although  the  Candidate  Multiple  is  useful  for  predicting 
midshipmen  performance,  it  has  not  included  selection  factors  concerned  with  officer  potential. 

Objective 

The  primary  objective  of  this  effort  is  to  expand  the  scope  of  the  USNA  selection  system  to 
include  the  prediction  of  officer  performance. 

Approach 

The  overall  validation  strategy  involved  four  major  components:  (1)  development  of  an 
officer  performance  criterion,  (2)  evaluation  of  the  Academy’s  current  selection  scores  for 
predicting  that  criterion,  (3)  development  and  validation  of  new  scales  to  predict  that  criterion 
using  instruments  currently  in  the  Academy’s  selection  system,  and  (4)  determination  of  the 
relationship  between  predictors  of  officer  performance  and  measures  of  Academy  success. 

Results  and  Conclusions 

1.  Although  prior  research  has  shown  that  most  ratings  on  officer  fitness  reports  do  not 
provide  adequate  discrimination,  there  is  sufficient  variability  in  early  promotion 
recommendations  to  provide  meaningful  differentiation  among  officers. 

2.  Although  the  operational  selection  measure  (CM)  is  not  related  to  officer  performance 
as  measured  by  Recommendation  for  Early  Promotion  (REP),  two  of  its  components, 
Extracurricular  Activities  (COMPECA)  and  Recommendations  (RECS),  individually  exhibit 
significant  relationships. 

3.  Performance  as  a  USNA  midshipman  is  significantly  related  to  officer  performance  as 
measured  by  the  REP. 

4.  Predictors  empirically  derived  from  applicant  data  reveal  significant  relationships  with 
the  officer  performance  criterion.  These  relationships  are  noteworthy  since  there  is  considerable 
time  between  USNA  application  and  the  cumulative  measurement  of  officer  performance. 

5.  Although  the  experimental  predictors  and  composite  show  low  positive  relationships 
with  USNA  performance,  it  appears  that  their  use  will  not  adversely  affect  performance. 
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Recommendations 


Based  on  these  results,  it  is  recommended  that:  (1)  the  experimental  measure  of  officer 
performance  (REP)  be  further  evaluated  by  examining  its  relationship  to  measures  or  status 
variables  concerned  with  actual  promotion  and  promotion  rates,  (2)  the  relationship  between 
officer  retention  and  the  experimental  criterion  (REP)  should  be  explored,  and  (3)  an  Officer 
Potential  Composite  (OPC),  consisting  of  RECS,  COMPECA,  and  an  empirically  derived 
predictor,  should  be  considered  for  experimental  integration  into  the  current  USNA  selection 
system  and  evaluated  further. 
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INTRODUCTION 


Background  and  Problem 

Each  year  some  1,350  young  men  and  women  are  selected  for  appointment  to  the  United 
States  Naval  Academy  (USNA)  from  a  pool  of  approximately  15,000  applicants.  To  minimize  both 
training  and  personnel  costs,  it  is  important  to  select  those  applicants  with  the  highest  potential. 
Current  selection  procedures  are  quite  successful  in  identifying  applicants  with  the  greatest 
potential  for  success  at  the  USNA.  These  procedures,  which  include  a  variety  of  selection  factors 
related  to  academy  success  have  been  combined  into  a  single  composite  measure,  the  Candidate 
Multiple,  to  serve  as  the  primary  tool  for  evaluating  candidates.  Although  the  Candidate  Multiple 
is  useful  for  predicting  successful  midshipmen  performance,  it  has  not  included  selection  factors 
concerned  with  officer  potential.  While  this  deficiency  in  the  Candidate  Multiple  has  been  clear 
from  the  inception,  the  necessary  longitudinal  data  have  only  recently  become  available  for  relating 
applicant  data  to  post-commissioning  officer  performance. 

Objective 

The  primary  objective  of  this  effort  is  to  expand  the  scope  of  the  USNA  selection  system  to 
include  the  prediction  of  officer  performance.  The  major  steps  in  pursuing  this  objective  are  to  ( 1 ) 
develop  and  assign  officer  performance  scores  from  existing  officer  fitness  reports,  (2)  identify 
relevant  individual  difference  variables  that  demonstrate  stable  relationships  with  such  measures 
of  officer  performance,  and  (3)  evaluate  the  impact  of  such  officer  potential  indicators  on  USNA 
predictors  and  criteria. 


APPROACH 

Tac  overall  validation  strategy  involved  four  major  components:  (1)  development  of  an 
officer  performance  criterion,  (2)  evaluation  of  the  Academy’s  current  selection  scores  for 
predicting  that  criterion,  (3)  development  and  validation  of  new  scales  to  predict  that  criterion 
using  instruments  currently  in  the  Academy’s  selection  system,  and  (4)  determination  of  the 
relationship  between  predictors  of  officer  performance  and  measures  of  Academy  success. 

Subjects 

Two  groups  of  subjects  were  used.  The  first  included  all  Navy  officers  commissioned  from 
the  USNA  classes  of  1979  through  1982.  These  3,309  subjects  were  used  to  develop  and  validate 
experimental  criteria  and  predictors  of  officer  potential. 

These  officers  were  among  the  most  recent  to  have  (1)  at  least  4  years  of  commissioned 
service  with  corresponding  officer  fitness  records,  (2)  USNA  selection  and  performance  scores 
available,  and  (3)  item  response  data  for  instruments  used  in  the  USNA  selection  program.  For  the 
most  part,  the  officers  whose  fitness  reports  are  included  here  were  still  on  active  duty  at  the  grade 
levels  considered.  Several  year  groups  were  included  to  help  ensure  that  results  obtained  would  be 
stable  across  classes. 

Table  1  presents  the  number  of  individuals  with  useable  data  in  each  of  the  4  year  groups, 
broken  out  by  warfare  specialty. 
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Table  1 


Sample  Sizes  Within  Warfare  Specialty  for 
theUSNA  1979-1982  Classes 


USNA  Class 


Warfare  Specialty 

1979 

1980 

1981 

1982 

Total 

Surface 

212 

145 

194 

208 

759 

Sub 

129 

165 

190 

161 

645 

Air 

252 

266 

288 

308 

1,114 

No  specialty 

206 

195 

168 

222 

791 

Total 

799 

771 

840 

899 

3,309 

The  second  sample  included  4,823  midshipmen  from  the  classes  of  1987  through  1990. 
While  these  recent  classes  have  insufficient  or  no  officer  performance  data,  they  were  used  to  help 
evaluate  the  impact  these  experimental  predictors  may  have  on  Academy  performance  if  used  in 
selection.  If  negative  relationships  were  observed  between  potential  predictors  and  academy 
criteria,  it  would  be  difficult  to  consider  their  use  in  selection.  This  undesirable  outcome  would 
occur  if  midshipmen  scoring  high  on  an  experimental  predictor  had  lower  Academic  Quality  Point 
Ratio  (AQPR)  or  Military  Quality  Point  Rating  (MQPR)  scores.  These  classes  were  the  most 
current  available  with  at  least  1  year  of  USNA  academic  and  military  grades.  The  numbers  of 
individuals  available  for  analyses  are  presented  in  Table  2. 

Table  2 

Sample  Sizes  for  the  USNA  1987-1990  Classes 


USNA  Classes 


Sample 

1987 

1988 

1989 

1990 

Total 

Applicant 

5,773 

4,924 

5,294 

5,282 

21,273 

Class 

1,328 

1,334 

1,149 

4,823 

Predictors 

The  measures  which  could  be  used  as  possible  predictors  of  officer  potential  ate  presented 
below  under  “operational”  or  “experimental”  predictors.  The  eight  variables  listed  under 
operational  predictors  are  currently  used  in  the  selection  program.  The  two  predictors  described  as 
experimental  were  specifically  constructed  to  predict  officer  performance.  Both  scales  were 
derived  from  items  included  in  currently  administered  instruments. 
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Operational 


1.  Scholastic  Aptitude  Test-Verbal  (SAT-V)  or  its  empirically-derived  American 
College  Test  (ACT)  equivalent.  This  score  represents  the  verbal  aptitude  of  the  individual  as 
measured  in  a  national  competitive  testing  program  designed  for  college  admissions  and 
scholarship  awards.  A  minimum  score  of  520  is  required  for  Naval  Academy  qualification,  with 
waivers  granted  in  exceptional  cases. 

2.  Scholastic  Aptitude  Test-Mathematics  (SAT-M)  or  its  empirically-derived  ACT 
equivalent.  This  score  parallels  the  SAT-V  except  that  it  represents  an  individual’s  quantitative 
aptitude.  Since  the  curriculum  at  the  Naval  Academy  is  heavily  weighted  with  quantitatively- 
oriented  courses,  the  minimum  qualifying  SAT-M  score  is  600.  Waivers  are  occasionally  allowed. 

3.  Rank  in  Class  (R/C).  This  is  a  standardized  score  (M  =  500,  SD  =  i00),  which  is  based 
on  an  individual’s  high  school  rank.  It  ranges  from  200  to  800. 

4.  Recommendation s  (RECS).  This  is  a  score  based  on  high  school  officials’  estimates  of 
the  applicant’s  potential  for  success  as  a  naval  officer.  The  officials  (usually  one  English  and  one 
Mathematics  teacher)  evaluate  the  candidate  on  communication  skills,  interpersonal  relations, 
personal  conduct,  and  leadership  potential.  An  objective  score  derived  by  summing  these 
evaluations,  ranges  from  0  to  1,000,  with  no  minimum  qualifying  score  required. 

5.  Extracurricular  Activities  (COMPECA).  This  is  a  score  based  on  the  Candidate 
Activities  Record  (CAR),  a  rather  extensive  application  form  that  covers  participation  in  both 
athletic  and  non-athletic  extracurricular  activities  during  high  school.  A  rationally-derived  scoring 
system  is  used  to  compute  the  COMPECA  score  that  ranges  from  300  to  800.  There  is  no  minimum 
qualifying  score. 

6.  Career  Interest  Scale  (CIS).  The  CIS  consists  of  items  from  the  Strong-Campbell 
Interest  Inventory  (SCII),  a  commercially-available  career  guidance  instrument,  which  has  been 
keyed  to  differentiate  between  high-  and  low-tenure  Naval  Academy  officers  (Neumann  & 
Abrahams,  unpublished  report,  23  September  1982). 

7.  Technical  Interest  Scale  (TIS).  This  scale  consists  of  items  from  the  SCO  which  have 
been  keyed  to  identify  Academy  applicants  with  high  interest  in  a  technically-oriented  curriculum 
(Neumann  &  Abrahams,  1974). 

8.  Candidate  Multiple  (CM).  The  Candidate  Multiple  is  an  empirically-developed 
composite  which  weights  SAT/ACT  scores,  high  school  performance,  extracurricular  activities, 
letters  of  recommendation,  and  vocational  interest  scales  to  predict  a  combination  of  academic 
performance,  military  performance,  and  disenrollment  (Alf,  Neumann,  &  Mattson,  1988). 
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Experimental 


1.  Activities  Performance  Scale  (APS).  In  an  attempt  to  improve  over  the  rational  score, 
COMPECA,  the  CAR  was  analyzed  to  identify  those  extracurricular  items  that  were  related  to 
officer  performance  for  inclusion  in  this  scale. 

2.  Interest  Performance  Scale  (IPS).  The  SCO  was  analyzed  to  identify  those  items  that 
were  related  to  officer  performance  for  inclusion  in  this  scale. 

Criteria 

Two  main  types  of  criteria  were  used  in  this  research:  (1)  measures  of  officer  performance 
and  (2)  measures  of  USNA  success.  Officer  performance  criteria  were  needed  to  develop 
experimental  predictors  to  be  considered  for  inclusion  in  the  CM.  The  USNA  criteria  were  needed 
to  assess  the  impact  that  these  predictors  might  have  on  Naval  Academy  performance  if  used  for 
midshipmen  selection. 

Officer  Performance 

Since  there  is  no  generally  accepted  single  measure  of  officer  performance  for  use  in  this 
research,  it  became  necessary  to  examine  a  variety  of  alternative  measures.  These  measures  were 
derived  from  performance  data  already  existing  in  officer  personnel  records  rather  than  from 
specially-administered  “research  only”  performance  evaluations.  The  operational  instruments  have 
the  advantage  of  (1)  providing  a  cumulative  record  of  performance,  (2)  including  performance 
information  from  a  variety  of  career  points,  jobs,  and  reporting  seniors,  and  (3)  requiring  no  added 
expense  or  intrusion. 

By  contrast,  a  “research  instrument”  developed  to  measure  officer  performance  would  have 
the  disadvantage  of  (1)  requiring  considerable  personnel  development  time  and  cost,  (2)  providing 
a  measure  at  only  one  point  in  time,  from  one  reporting  senior,  for  one  job,  and  (3)  only  being 
available  for  officers  whose  supervisors  agreed  to  participate.  Since  the  fitness  reports  are  routinely 
used  in  officer  evaluations,  they  also  afford  a  higher  level  of  acceptability  among  the  officer 
community  than  would  a  “research  only”  measure.  Examination  of  fitness  report  data,  directed  at 
developing  a  useful  performance  criterion  score,  will  be  described  in  the  Analyses  section. 

USNA  Performance 

The  measures  of  midshipmen  performance  deemed  most  relevant  to  assessing  experimental 
predictor  impact  are  described  below: 

1 .  Academic  Quality  Point  Ratio  (AQPR).  The  AQPR  used  in  this  study  is  the  cumulative 
grade-point  average  earned  by  a  midshipman  after  4  years  at  the  Academy  for  the  officer  sample 
and  after  1  year  for  the  midshipmen  sample.  If  a  midshipman  disenrolled,  the  last  available  AQPR 
was  used. 

2.  Military  Quality  Point  Ratine  (MQPR).  The  MQPR  is  a  composite  rating  assigned  at 
the  end  of  every  semester  to  each  midshipman.  It  is  a  weighted  composite  based  on  grades  earned 
in  physical  education,  professional  competency  review,  military  performance,  conduct,  and 
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professional  courses.  This  rating  is  of  special  importance  since  research  conducted  at  the  Academy 
(Howland,  1970) 1  and  more  recently  at  Navy  Personnel  Research  and  Development  Center 
(NAVPERSRANDCEN)  has  shown  that  a  similar  measure  is  related  to  later  officer  performance. 
A  cumulative  MQPR  was  computed  for  each  midshipman,  based  on  the  4  years’  ratings  at  the 
Academy  for  the  officer  sample  and  after  1  year  for  the  midshipmen  sample.  As  with  the  AQPR, 
if  a  midshipman  disenrolled,  the  last  available  MQPR  was  used. 

3.  All  Disenrollment  (ALL  DIS).  Since  any  decrease  in  the  overall  disenrollment  rate 
would  be  desirable,  the  prediction  of  this  criterion  was  examined.  This  measure  includes  all 
midshipmen  who  separate  from  the  Academy  either  voluntarily  or  involuntarily,  with  the  exception 
of  medical  discharges.  The  latter  are  not  included  since  it  seems  highly  unlikely  that  such 
separations  would  be  predictable  from  the  variables  being  investigated  here.  In  the  most  recent 
graduating  class  (1987),  about  19  percent  of  the  midshipmen  disenrolled  voluntarily  or 
involuntarily  within  the  first  2  years,  and  approximately  23  percent  disenrolled  by  the  end  of  the 
4-year  program. 

Development  of  the  officer  potential  criterion  began  with  a  thorough  review  of  information 
routinely  included  in  each  fitness  report.  Based  on  this  review,  three  variables  or  rating  dimensions 
were  identified  that  potentially  provide  (1)  comprehensive  evaluation,  (2)  periodic  ratings,  and  (3) 
relatively  complete  data. 

1  .  Command  Desirabilitv--a  reporting  senior’ s  rating  of  the  desirability  of  an  officer  being 
under  his/her  command  in  a  “command  assignment.” 

2.  Mission  Contribution-Evaluation-a  reporting  senior’s  evaluation  of  “the  officer’s 
contribution  to  the  unit’s  mission  while  effectively  integrating  the  man  and  the  mission.” 

Both  the  command  desirability  and  mission  contribution -evaluation  ratings  are  made  on  a 
percentile  scale  that  ranges  from  the  top  1  percent  to  the  bottom  30  percent  with  five  intermediate 
steps. 


3.  Recommendation  for  Promotion-a  reporting  senior’s  recommendation  of  the 
individual  for  (1)  early  promotion,  (2)  regular  promotion,  or  (3)  no  promotion. 

Analyses 

Analyses  were  directed  at  four  primary  objectives:  (1)  officer  potential  or  performance 
criterion  development,  (2)  validation  of  operational  selectors,  (3)  development  of  experimental 
predictors  of  officer  performance,  and  (4)  evaluation  of  experimental  predictors  on  current  USNA 
classes. 


'Howland,  R.  W.  (July  1970).  Recruitment  research  study  group  study  plan  (Memorandum  to 
Superintendent:  U.S.  Naval  Academy). 
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Criterion  Development 


For  each  of  the  three  potential  criteria,  a  summary  score  across  an  officer’s  fitness  records 
was  computed.  Fitness  reports  were  considered  qualified  and  included  in  this  summary  only  if  (1) 
they  were  based  on  ensign  (ENS),  lieutenant  junior  grade  (LTJG),  and  lieutenant  (LT) 
performance,  (2)  the  reporting  senior  indicated  that  ratings  were  based  on  close  observation,  and 
(3)  the  reporting  senior  was  simultaneously  rating  other  officers  in  his/her  command  (i.e.,  either  a 
“periodic”  or  “detachment  of  reporting  senior”  occasion).  The  latter  restriction  served  to  eliminate 
fitness  reports  where  a  reporting  senior  evaluated  only  one  officer  who  is  being  detached,  unlike 
the  periodic  ratings  where  all  officers  are  rated.  Application  of  these  three  criteria  resulted  in  the 
exclusion  of  a  small  portion  of  all  fitness  reports  available. 

Inspection  of  the  criterion  distributions  revealed  a  large  number  of  individuals  receiving  top 
ratings.  Since  this  kind  of  skewed  distribution  could  undermine  the  usefulness  of  a  rating,  a 
summary  score  was  computed  for  each  factor  to  indicate  the  proportion  of  occasions  on  which  the 
officer  received  the  highest  possible  rating.  A  score  of  1.00  thus  indicates  that  the  officer  received 
the  highest  rating  (e.g.,  top  1%  or  recommended  for  early  promotion)  for  all  qualified  fitness 
reports,  while  a  score  of  0.00  indicates  that  he  or  she  never  received  a  top  rating.  Using  these 
computational  methods,  over  58  percent  of  individuals  received  a  score  of  1.00  on  Command 
Desirability  and  over  61  percent  received  such  a  score  on  Mission  Contribution.2  Clearly,  these 
variables  showing  such  large  percentages  with  maximum  scores  would  not  provide  the 
differentiation  necessary  to  serve  as  criteria. 

By  contrast,  only  26  percent  were  recommended  for  early  promotion  on  all  qualified  fitness 
records  (i.e.,  26%  received  the  highest  score  of  1.00  on  the  Recommended  for  Promotion  score). 
Taken  together,  these  data  indicate  that  this  score  offers  the  greatest  potential  of  all  the  i  neasures 
for  use  as  a  performance  criterion.  In  addition  to  its  psychometric  properties,  a  recommendation 
for  promotion  is  considered  an  important  means  of  identifying  superior  officers:  As  indie  eted  in 
the  Navy  Military  Personnel  Command  Instruction  1611.1, 12  May  1981: 

A  recommendation  for  early  promotion  is  appropriate  for  outstanding  officers  who 
are  considered  by  their  reporting  seniors  to  merit  promotion  ahead  of  their 
contemporaries.  It  is  entirely  acceptable  for  such  nomination  to  be  made  regardless 
of  the  time  in  grade  or  promotional  eligibility,  for  this  procedure  serves  to  identify 
our  “head-and-shoulders”  type  performers. 

Scores  based  on  the  “recommended  for  promotion”  ratings  were  generated  as  described 
above  for  all  officers  in  the  classes  of  1979-1982.  Means  were  computed  separately  by  class  for 
each  rank  within  warfare  specialty  and  are  reported  in  Table  3.  Large  differences  in  mean  criterion 
scores  were  observed  for  individuals  of  different  grades,  warfare  specialties,  and  year  groups.  Of 


2NAVMILPTiRSCOM  Instruction  1611.1,  NMPC-323  recently  (Feb  87)  formalized  the  introduction  of  two 
additional  rfficer  performance  dimensions  (i.e.,  tactical  proficiency  and  leadership).  Since  these  scales  were 
recently  introduced,  ratings  are  only  available  on  a  limited  sample.  A  preliminary  analysis  of  the  ratings 
assigned  to  these  two  evaluations  revealed  similar  lack  of  variability.  For  example,  79  percent  of  all  officers 
received  the  highest  possible  score  (i.e.,  top  1%)  on  the  leadership  evaluations.  Tactical  proficiency  showed 
even  less  variability,  with  85  percent  of  all  officers  receiving  the  highest  score. 
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Table  3 


Mean  REP  Score  by  Grade  Within  Warfare  Specialty  for  the 
USNA  1979-1982  Classes 


Warfare  Specialty 

Grade 

1979 

USNA  Class 

1980  1981 

1982 

Total 

Surface 

LT 

.576 

.652 

.528 

.590 

LTJG 

.414 

.552 

.483 

.466 

FNS 

.319 

.216 

.311 

.291 

Sub 

LT 

.671 

.674 

.703 

.677 

.683 

LTJG 

.448 

.455 

.528 

.524 

.492 

ENS 

.243 

.283 

.276 

.248 

.266 

Air 

LT 

.481 

.468 

.461 

.489 

LTJG 

.249 

.217 

.298 

.268 

.259 

ENS 

.199 

.316 

.151 

Not  qualified 

LT 

.475 

.492 

.584 

.491 

.508 

LTJG 

.367 

.385 

.468 

.435 

.412 

ENS 

.197 

.370 

.327 

.238 

.286 

Note.  Since  officers  were  followed  longitudinally,  the  same  officer  may  be  represented  at  each  rank 
(i.e.,  ENS,  LTJG,  LT). 


the  three  classification  variables,  differences  between  grades  are  most  pronounced.  While  sizable 
differences  exist  between  class  and  specialty,  they  are  somewhat  less  dramatic.  As  might  be 
expected,  higher  ranking  officers  obtain  higher  scores  than  lower  ranking  officers. 

The  magnitude  of  these  differences  argued  against  the  pooling  of  all  individuals  when  using 
raw  criterion  scores.  To  eliminate  these  rating  differences  and  permit  the  pooling  of  scores, 
standard  scores  (M  =  50,  SD  =  10)  were  derived  for  each  subgroup  defined  in  Table  3.  For  example, 
within  the  surface  community  all  fitness  reports  completed  for  those  commissioned  in  1981  while 
they  were  LTJGs  formed  one  of  the  48  subgroups  within  which  scores  were  standardized.  In  this 
way,  three  standard  scores  were  computed  for  each  officer,  one  based  on  ENS  fitness  reports,  one 
based  on  LTJG  fitness  reports  and  one  based  on  LT  fitness  reports.  Each  of  these  scores  indicates 
the  officer’s  position  relative  only  to  his  same-rank  peers  within  his  specialty  and  class.  The  three 
scores  were  then  weighted  by  the  number  of  fitness  reports  involved  and  summed  to  generate  a 
single  standard  score  that  reflects  the  officer’s  complete  fitness  report  history.  This  weighted  sum 
of  standardized  scores  is  referred  to  as  the  Recommended  for  Early  Promotion  (REP)  criterion. 


Validation  of  Operational  Selectors 

As  indicated  earlier,  the  operational  predictors  had  been  developed  for  predicting  success  at 
USNA  and  are  currently  used  in  midshipmen  selection.  To  determine  their  relationship  to  the  REP, 
validity  coefficients  were  computed  for  the  seven  predictors  that  comprise  the  selection  composite 
and  were  also  computed  for  the  selection  composite  itself  (CM).  In  addition,  the  two  USNA 
performance  criteria,  AQPR  and  MQPR  were  correlated  with  the  REP.  All  correlations  with  REP 
were  computed  for  the  total  sample  as  well  as  for  each  warfare  specialty. 

Development  of  Experimental  Predictors 

The  CAR,  a  basis  for  one  of  the  experimental  scales  for  predicting  officer  performance, 
requires  each  applicant  to  indicate  participation,  honors,  and  awards  for  a  wide  variety  of  athletic 
and  non-athletic  activities  for  each  year  of  high  school.  To  enhance  the  stability  of  these  individual 
CAR  items  based  on  each  year’s  participation,  the  separate  responses  for  the  10th,  1 1th,  and  12th 
grade  were  combined  into  a  single  variable.  In  this  way,  each  “summary  response”  reflects  the 
entire  high  school  record  for  the  activity,  award,  or  honor. 

To  develop  and  evaluate  experimental  scales  for  the  SCII  and  the  CAR,  two  separate 
subsamples  were  needed,  one  for  item  analysis  and  one  for  cross-validation.  One  subsample  was 
formed  by  randomly  selecting  approximately  40  percent  of  the  total  group  for  cross-validation 
(N  =  927).  Of  the  remaining  midshipmen  (N  =  1 ,324),  only  those  scoring  in  the  upper  or  lower  one- 
third  of  the  criterion  distribution  were  used  for  key-  construction  (N  =  888). 

To  construct  the  scale  on  the  CAR  items,  the  proportion  endorsing  each  summary  response 
was  computed  separately  for  the  high  and  low  performers.  For  each  activity,  honor,  or  award  the 
difference  between  these  proportions  was  computed.  In  this  way,  those  16  item  responses  with 
differences  greater  than  eight  percent  were  retained  for  inclusion  in  an  experimental  scale  called 
the  Activities  Performance  Scale  (APS).  Similarly,  an  experimental  SCO  scale,  the  Interest 
Performance  Scale  (IPS)  consisting  of  40  item  responses  with  differences  greater  than  12  percent 
was  developed.  The  same  key-  construction  and  cross-validation  groups  were  used  for  the  IPS  as 
were  used  for  the  APS. 

Finally  the  APS  and  the  IPS  Scales  were  evaluated  for  their  prediction  of  officer  performance 
over  and  above  that  afforded  by  operational  measures. 

Evaluation  of  Experimental  Predictors  on  the  Most  Recent  Classes 

Although  it  is  not  possible  to  validate  the  experimental  officer  performance  predictors  against 
an  officer  performance  criterion  on  the  most  recent  USNA  classes,  it  is  possible  to  assess  the 
impact  of  these  potential  predictors  against  USNA  performance  (AQPR,  MQPR,  and  ALL  DIS). 
For  example,  if  one  or  both  of  these  scales  was  a  useful  predictor  of  REP  but  had  a  negative 
relationship  with  USNA  performance,  its  usefulness  would  be  limited.  These  relationships  were 
explored  for  the  combined  1987-1990  classes. 


RESULTS  AND  DISCUSSION 


Validity  of  Operational  Selectors 

Table  4  presents  the  uncorrected  zero-order  validity  coefficients  of  the  operational  selectors 
with  the  standardized  experimental  performance  score  for  the  combined  sample  as  well  as  the 
warfare  specialties. 

Overall,  the  operational  selectors  show  varying  relationships  with  the  REP  criterion.  The 
three  academically  oriented  selection  scores  (SAT-Verbal,  SAT-Math,  and  Rank-in  Class)  yield 
correlations  of  -.01,  -.06,  and  .01  with  REP  in  the  total  sample.  Of  the  remaining  predictors,  only 
the  non-academically  oriented  RECS  and  COMPECA  showed  useful  relationships  with  the  officer 
performance  criterion  (i.e.,  .09  and  .13  respectively). 

The  RECS  score,  while  showing  a  promising  relationship  with  REP  is  based  on  an  earlier 
recommendations  form  than  is  currently  being  used'*.  As  a  consequence,  the  currently  used  RECS 
score  may  be  considered  parallel  but  not  equivalent. 

The  single  most  valid  predictor  of  REP  is  the  COMPECA  score  (r  =  .13)  derived  from  the 
CAR.  While  there  have  been  a  few  minor  revisions  of  questions  on  this  form,  they  are  unlikely  to 
affect  the  rank  order  of  applicants,  since  they  concern  less  popular  activities. 

The  CM,  a  weighted  combination  of  operational  selectors  designed  to  predict  USNA 
performance  measures,  yields  a  validity  of  .01  for  predicting  REP  in  the  total  sample  (Table  4).  The 
discrepancy  between  the  CM’s  validity  of  .01  and  the  higher  validities  shown  for  the  COMPECA 
and  RECS  components  is  a  result  of  the  relatively  low  weight  given  to  COMPECA  and  RECS  in 
the  CM. 

Validities  of  the  operational  selectors  for  each  warfare  specialty  are  also  shown  in  Table  4. 
The  pattern  of  validities  closely  parallels  that  observed  on  the  total  sample.  The  COMPECA  and 
RECS  tended  to  show  the  highest  relationship. 

These  results  taken  together  suggest  that  the  academically-oriented  predictors  (i.e.,  SAT, 
R/C)  are  not  useful  in  the  early  identification  of  officer  potential.  At  first  glance,  this  may  suggest 
that  academic  ability  is  completely  unrelated  to  officer  performance.  However,  it  should  be 
emphasized  that  those  who  are  admitted  and  ultimately  commissioned  are  highly  selected  on 
academic  measures.  On  the  other  hand,  the  nonacademic  predictors  appear  to  be  mildly  related  and 
may  benefit  from  additional  analyses  focussed  on  identifying  their  most  valid  components. 


■^Improvements  made  in  the  new  form’s  scale  and  items  increased  its  relationship  to  USNA 
military  performance. 
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Table  4 


Validity  of  USNA  Selection  and  Performance  Scores  to  Predict 
REP  for  the  Combined  USNA  1979-1982  Classes 


Warfare  Specialty 

Scores 

Surface 
(N  =  501) 

Sub 

(N  =  579) 

Air 

(N  =  685) 

Not 

Qualified 
(N  =  486) 

Total 

(N  =  2251) 

Selection 

SAT-V 

.05 

-.08* 

.05 

-.03 

-.01 

SAT-M 

-.03 

-.08* 

-.09 

-.11** 

-.06* 

R/C 

-.03 

-.04 

.05 

-.05 

.01 

RECS 

.10** 

.04 

.14** 

.03 

.09* 

COMPECA 

.16** 

.07* 

.11** 

.19** 

.13** 

TIS 

-.10* 

-.04 

-.09** 

-.03 

-.07 

CIS 

.02 

.03 

.02 

.00 

.02 

CM 

.04 

-.02 

.03 

.01 

.01 

USNA  Performance 

AQPR 

.18** 

.16** 

.28** 

.03 

.15** 

MQPR 

.30** 

.26** 

.34** 

.15** 

.26** 

*p  <  .05. 

**p  <  .01. 


Although  the  academically-oriented  selection  scores  show  little  relationship  to  officer 
performance,  the  measure  of  actual  USNA  academic  performance,  AQPR,  correlates  .15  with  the 
REP,  as  shown  in  Table  4.  Further,  MQPR,  a  measure  of  military  performance  as  a  midshipmen, 
is  even  more  highly  related  to  later  performance  (r  =  .26)  as  an  officer.  While  clearly  not  useful 
as  selection  variables,  it  is  reassuring  to  note  that  these  measures  of  performance  in  training  are 
related  to  the  newly-developed  officer  performance  measure  (REP). 

Experimental  Predictors 

Table  5  presents  means,  standard  deviations,  and  correlations  for  the  key  construction  and 
cross-validation  samples  used  in  constructing  and  validating  the  first  of  two  experimental  scales 
(i.e.,  the  IPS  Scale).  A  statistically  significant  correlation  of  .14  between  the  IPS  Scale  and  REP 
was  obtained  for  the  cross-validation  sample.  Similarly,  the  APS  scale  based  on  the  CAR,  as  seen 
in  Table  6,  showed  a  slightly  lower  but  significant  relationship  as  indicated  by  its  validity  of  .12. 
By  contrast,  the  operational  score  derived  from  the  CAR  (i.e.,  the  COMPECA)  yields  a  validity  of 
.16  on  the  same  cross-validation  sample. 
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Table  5 


Means,  Standard  Deviations,  and  Correlations  of  the  Interest 
Performance  Scale  for  the  Prediction  of  REP  Across 


Warfare  Specialties 

Sample 

N 

Mean 

S.D. 

Correlation 
With  REP 

Kev-construction 

High-criterion 

454 

105.46 

4.77 

Low-criterion 

434 

101.85 

4.97 

Total 

888 

103.69 

5.19 

.34 

Cross-validation 

927 

103.67 

5.17 

.14* 

*p  <  .01. 

Table  6 


Means,  Standard  Deviations,  and  Correlations  of  the  Activities 
Performance  Scale  for  the  Prediction  of  REP  Across 
Warfare  Specialties 


Sample 

N 

Mean 

S.D. 

Correlation 

With  REP 

Kev-construction 

High-criterion 

454 

98.42 

2.12 

Low-criterion 

434 

97.98 

1.85 

Total 

888 

98.20 

2.00 

.11 

Cross-validation 

927 

98.25 

1.98 

.12* 

*p  <  .05. 


Considering  the  validities  of  all  operational  and  experimental  predictors,  four  appear  to  offer 
potential  for  predicting  officer  performance.  These  include  the  operational  RECS  and  COMPECA 
in  addition  to  the  experimental  EPS  and  APS  Scales.  Since  the  experimental  APS  is  slightly  less 
valid  and  uses  significantly  fewer  items  than  COMPECA,  the  operational  scale  was  selected  along 
with  RECS  and  IPS  for  inclusion  in  a  composite  score  to  predict  officer  performance. 


Table  7  contains  the  results  of  combining  these  three  variables  into  a  single  Officer  Potential 
Composite  (OPC)  score.  Correlations  between  the  composite  and  performance  are  shown  for  each 
warfare  specialty  as  well  as  the  total  sample.  The  composite’s  multiple  R  of  .21,  when  corrected 
for  “shrinkage”  produced  a  validity  of  .20  for  the  total  group.  When  used  with  a  highly  favorable 
selection  ratio  (i.e.,  approximately  one  selectee  per  seven  qualified  applicants),  a  correlation  of  this 
magnitude  suggests  a  potentially  useful  predictor  of  officer  performance.  Further,  when  evaluating 
this  relationship,  it  should  be  kept  in  mind  that  correlations  as  high  as  this  over  a  comparable  period 
of  time  arc  rarely  reported  in  the  personnel  selection  literature.  The  correlations  of  this  composite 
within  warfare  specialty  ranged  from  a  high  of  .25  for  Surface  Warfare  officers  to  a  low  of  .13  for 
the  Submarine  Warfare  specialty. 


Table  7 

Validity  of  IPS,  COMPECA,  RECS,  and  an  Officer  Potential  Composite 
to  Predict  REP  for  the  Combined  USNA  1979-1982 
Classes  by  Warfare  Specialty 


Warfare  Specialty  Qualified 

Not 


Scores 

Surface 
(N  =  221) 

Sub 

(N  =  579) 

Air 

(N  =  685) 

Qualified 
(N  =  486) 

Total 

(N=  1971) 

IPS 

.14* 

.18** 

.15** 

.05 

.14** 

RECS 

.17* 

.02 

.16** 

.10 

.11** 

COMPECA 

.17* 

.06 

.14** 

.27** 

.16** 

OPC3 

.25** 

.13* 

.21** 

.24** 

.21** 

*p  <  .05. 

**p  <  .01. 

aThe  OPC  score  is  a  composite  of  the  IPS,  RECS,  and  COMPECA  to  predict  the  REP. 

To  assess  the  impact  of  the  experimental  predictor  of  officer  performance,  DPS,  and  the  OPC 
composite  on  USNA  midshipmen  performance,  their  relationships  to  USNA  performance  criteria 
are  shown  in  Table  8.  To  provide  baseline  information  for  purposes  of  comparison,  validities  of  the 
operational  selection  variables  are  also  provided. 

Although  the  composite  was  not  developed  to  predict  USNA  performance,  the  correlations 
shown  in  Table  8  reveal  a  significant  positive  relationship  with  MQPR  and  ALL  DIS.  These  data 
suggest  there  is  little  risk  that  its  use  will  adversely  affect  midshipmen  performance. 
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Table  8 


Validity  of  USNA  Selection  Scores  and  an  Officer  Potential 
Composite  to  Predict  USNA  Performance  for  the  1987-1990  Classes 
at  the  End  of  Plebe  Year 


USNA  Performance 

Scores 

AQPR 

MQPR 

ALL  DIS 

Selection 

SAT-V 

.31** 
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.03* 

.03* 

.05 

CIS 

.09** 

.07** 

.03 

CM 

.50** 

.34** 

.09 

Experimental3 

IPS 

-.05** 

-.01 

.07 

OPCb 

-.01 

.07** 

.10* 

Note.  AQPR  and  MQPR  validities  are  Pearson  r’s  while  the  ALL  DIS  validities  are  biserial  r’s. 


*p  <  .05. 

**p<.01. 


aThe  Experimental  IPS  Scale  was  constructed  on  the  SCII  responses  across  Warfare  Specialties. 
bOPC  score  is  a  composite  of  the  IPS  and  the  operational  RECS  and  COMPECA  scores. 


CONCLUSIONS 

Preliminary  development  and  evaluation  of  an  experimental  measure  of  officer  performance 
as  well  as  experimental  predictors  of  this  performance  measure  suggest  the  following  conclusions. 

1.  Although  prior  research  has  shown  that  most  ratings  on  officer  fitness  reports  do  not 
provide  adequate  discrimination,  there  is  sufficient  variability  in  early  promotion  recommenda¬ 
tions  to  provide  meaningful  differentiation  among  officers. 

2.  Performance  as  a  USNA  midshipman  is  significantly  related  to  officer  performance  as 
measured  by  REP. 

3.  Although  the  operational  selection  measure  (CM)  is  not  related  to  the  REP,  two  of  its 
components,  COMPECA  and  RECS,  individually  exhibit  significant  relationships. 
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4.  Predictors  empirically  derived  from  applicant  data  reveal  significant  relationships  with 
the  officer  performance  criterion.  These  relationships  are  noteworthy  since  there  is  considerable 
time  between  USNA  application  and  the  cumulative  measurement  of  officer  performance. 

5.  Although  the  experimental  predictors  and  composite  show  low  positive  relationships 
with  USNA  performance,  it  appears  that  their  use  will  not  adversely  affect  performance. 

RECOMMENDATIONS 

Based  on  these  results,  it  is  recommended  that: 

1.  The  experimental  measure  of  officer  performance  (REP)  be  further  evaluated  by 
examining  its  relationship  to  measures  or  status  variables  concerned  with  actual  promotion  and 
promotion  rates. 

2.  The  relationship  between  officer  retention  and  the  experimental  criterion  (REP)  be 
explored. 

3.  The  OPC  be  considered  for  experimental  integration  into  the  current  USNA  selection 
system. 
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